Image processing device, image processing method, and storage medium storing image processing program

ABSTRACT

An image processing device includes a processor; and a memory to store the program which performs processing including: measuring a first texture amount indicating luminance variation in image regions, based on the image regions obtained by dividing the captured image and previously determining an image processing target region based on the first texture amount; judging whether the image processing on a current captured image is necessary based on luminance of the image processing target region in the current captured image; calculating a second texture amount indicating luminance variation in the image processing target region, and judging whether the image processing should be performed on the image processing target region or not based on the second texture amount; and performing the image processing on the image processing target region on which the judging based on the second texture amount is that the image processing should be performed.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of InternationalApplication No. PCT/JP2018/015383 having an international filing date ofApr. 12, 2018.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image processing device, an imageprocessing method, and an image processing program.

2. Description of the Related Art

When noise reduction processing is performed on captured image data(hereinafter referred to also as a “captured image”) acquired by animage capturing device, there are cases where a minute part of a subjectin the captured image is blurred, that is, image quality is degraded. Asa countermeasure against this problem, there has been proposed a devicethat judges whether the noise reduction processing is necessary or notbased on the luminance of the captured image and performs the noisereduction processing on the captured image when the noise reductionprocessing is judged to be necessary (see Patent Reference 1, forexample).

Patent Reference 1: Japanese Patent Application Publication No.2005-94087

However, the conventional device described above is unsuitable forsystems for monitoring the outdoor scene in the nighttime since thedevice performs the noise reduction processing when the captured imageis dark. For example, when the noise reduction processing is performedon a captured image including a road that is dark and airspace above theroad, a problem arises in that an image region of the road and an object(e.g., vehicle) on the road that have high monitoring priority isblurred.

SUMMARY OF THE INVENTION

An object of the present invention, which has been made to resolve theabove-described problem with the conventional technology, is to providean image processing device, an image processing method and an imageprocessing program that do not cause the blurring to an image regionhaving high monitoring priority even when the noise reduction processingis performed.

An image processing device according to an aspect of the presentinvention is a device that performs image processing for reducing noisein a captured image, including: a processor to execute a program; and amemory to store the program which, when executed by the processor,performs processing including measuring a first texture amountindicating luminance variation in a plurality of image regions, based onthe plurality of image regions obtained by dividing the captured imageand previously determining an image processing target region to be atarget of the image processing based on the first texture amount;judging whether the image processing on a current captured image isnecessary or not based on luminance of the image processing targetregion in the current captured image; calculating a second textureamount indicating luminance variation in the image processing targetregion for which the image processing is judged to be necessary in saidjudging based on the luminance, and judging whether the image processingshould be performed on the image processing target region or not basedon the second texture amount; and performing the image processing on theimage processing target region on which said judging based on the secondtexture amount is that the image processing should be performed.

An image processing method according to another aspect of the presentinvention is a method of performing image processing for reducing noisein a captured image, including: measuring a first texture amountindicating luminance variation in a plurality of image regions, based onthe plurality of image regions obtained by dividing the captured imageand previously determining an image processing target region to be atarget of the image processing based on the first texture amount;judging whether the image processing on a current captured image isnecessary or not based on luminance of the image processing targetregion in the current captured image; calculating a second textureamount indicating luminance variation in the image processing targetregion for which the image processing is judged to be necessary in thejudging based on the luminance, and judging whether the image processingshould be performed on the image processing target region or not basedon the second texture amount; and performing the image processing on theimage processing target region on which the judging based on the secondtexture amount is that the image processing should be performed.

According to the present invention, the blurring of an image regionhaving high monitoring priority can be prevented even when the noisereduction processing is performed.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become more fully understood from thedetailed description given hereinbelow and the accompanying drawingswhich are given by way of illustration only, and thus are not limitativeof the present invention, and wherein:

FIG. 1 is a block diagram schematically showing a configuration of animage processing device according to a first embodiment of the presentinvention;

FIG. 2 is a block diagram schematically showing a configuration of animage region calculation unit in FIG. 1 ;

FIG. 3 is a diagram showing an example of a captured image acquired byan image capturing device;

FIG. 4 is a flowchart showing a captured image region dividing operationand an image processing target region selection operation performed bythe image processing device according to the first embodiment;

FIG. 5 is a diagram showing an example of a divided region imageobtained by dividing the captured image into two image regions in theimage processing device according to the first embodiment;

FIG. 6 is a flowchart showing an operation of judging whether imageprocessing should be applied to each image processing target region ornot in the image processing device according to the first embodiment;

FIG. 7 is a block diagram schematically showing a configuration of animage processing device according to a second embodiment of the presentinvention;

FIG. 8 is a flowchart showing the operation of a subject positiondetection unit, a mask processing unit, and an image processing unit ofthe image processing device according to the second embodiment;

FIG. 9 is an explanatory diagram showing a process of transformingsubject coordinates in a camera coordinate system into subjectcoordinates in a camera image coordinate system;

FIG. 10(a) is a diagram showing a case where a subject exists in acaptured image acquired by the image capturing device in the nighttimewhen field illuminance is low (dark), and FIG. 10(b) is a diagramshowing a case where a mask range on which the image processing is notperformed is displayed in superposition on the divided region imageafter undergoing the processing by the image processing device accordingto the second embodiment; and

FIG. 11 is a diagram for explaining the size of a corrected mask rangeafter undergoing speed correction in a case where a moving speed of thesubject is known.

DETAILED DESCRIPTION OF THE INVENTION

Image processing devices, image processing methods and image processingprograms according to embodiments of the present invention will bedescribed below with reference to the accompanying drawings. Thefollowing embodiments are just examples and a variety of modificationsare possible within the scope of the present invention.

(1) First Embodiment (1-1) Configuration

FIG. 1 is a block diagram schematically showing a configuration of animage processing device 1 according to a first embodiment. FIG. 2 is ablock diagram schematically showing a configuration of an image regioncalculation unit 110 in FIG. 1 . The image processing device 1 is adevice capable of executing an image processing method according to thefirst embodiment. The image processing device 1 can also be a computercapable of executing an image processing program according to the firstembodiment.

As shown in FIG. 1 , the image processing device 1 includes a controlunit 10, a storage unit 20, an input interface 30 and an outputinterface 40. The control unit 10 includes the image region calculationunit 110, an image region readout unit 120, a luminance judgment unit130, a texture judgment unit 140 and an image processing unit 150. Asshown in FIG. 2 , the image region calculation unit 110 includes animage region dividing unit 111, texture measurement unit 112 and animage processing region selection unit 113.

The control unit 10 includes, for example, one or more processors forprocessing information, namely, one or more ICs (Integrated Circuits)for performing processing. Specifically, the control unit 10 includesone or more CPUs (Central Processing Units). The control unit 10 iscapable of controlling other hardware. The control unit 10 is capable ofexecuting software stored in the storage unit 20.

The control unit 10 stores information, data, signal values, variablevalues, etc. in a storage device 210, a memory 220, or a register orcache memory in the control unit 10. In FIG. 1 , each arrow connecting acomponent in the control unit 10 and the storage device 210 indicatesthat the component in the control unit 10 stores the result ofprocessing in the storage device 210 or that the component in thecontrol unit 10 loads information from the storage device 210. Arrowsconnecting components in the control unit 10 together indicate the flowof processing. Incidentally, while there are cases where imageinformation is transferred between components in the control unit 10 viathe memory 220 in FIG. 1 , arrows or connection lines connecting thememory 220 and the components in the control unit 10 are not shown inFIG. 1 .

A non-transitory computer-readable storage medium storing a program suchas an image processing program that implements the functions of thecontrol unit 10 may be stored in a memory (e.g., the storage unit 20) asa portable record medium such as a magnetic disk, a flexible disk, anoptical disk, a compact disc, a Blu-ray (registered trademark) disc or aDVD (Digital Versatile Disc).

The storage unit 20 is an information storage means for storinginformation. The storage unit 20 includes one or more storage devices210 and one or more memories 220. The storage device 210 includes a ROM(Read Only Memory), a flash memory or an HDD (Hard Disk Drive), forexample. The memory 220 includes a RAM (Random Access Memory), forexample. Incidentally, the storage unit 20 is not limited to theconfiguration shown in FIG. 1 .

The storage device 210 stores, for example, a texture threshold value211 as a predetermined first threshold value to be used when an imagecapture region is divided, image processing region information 212, apredetermined luminance judgment threshold value 213 to be used forjudging the necessity of image processing, and a texture judgmentthreshold value 214 as a predetermined second threshold value to be usedsimilarly for judging the necessity of image processing.

The storage device 210 may store an image processing program forimplementing the functions of the control unit 10. This image processingprogram is loaded into the memory 220, loaded in by the control unit 10,and executed by the control unit 10. In cases where the image processingdevice 1 is a computer, an OS (Operating System) may be stored in thestorage device 210. At least part of the OS is loaded into the memory220, and the control unit 10 executes the image processing program whileexecuting the OS.

The input interface 30 is a port to be connected to an image capturingdevice 3. The input interface 30 is an image input interface that takesa captured image acquired by the image capturing device 3 into the imageprocessing device 1. The captured image taken in by the input interface30 is stored in the memory 220, for example. The input interface 30 maybe equipped with a port to be connected to an input device (i.e., useroperation unit) such as a mouse, a keyboard, or a touch panel. The inputinterface 30 can be a port to be connected to a network such as a LAN(Local Area Network) (registered trademark), for example. The inputinterface 30 can be a USB (Universal Serial Bus) terminal. The inputinterface 30 can be a capture board capable of taking a video signalaccording to SDI (Serial Digital Interface), HDMI (registered trademark)(High Definition Multimedia Interface), VGA (Video Graphics Array), DVI(Digital Video Interface) or component into the image processing device1.

The output interface 40 is a port to which a cable of a display devicesuch as a display is connected. The output interface 40 is a USBterminal or an HDMI (registered trademark) terminal, for example. Thedisplay is an LCD (Liquid Crystal Display), for example.

(1-2) Operation

FIG. 3 is a diagram showing an example of a captured image acquired bythe image capturing device 3. In the example of FIG. 3 , image captureobjects include a road 301 and vehicles traveling on the road 301. Theimage capturing device 3 is, for example, a monitoring camera formonitoring road conditions. Monitoring targets (subjects) are vehiclestraveling on the road in a lower part of the captured image 300.Airspace is captured in an upper part of the captured image 300.

The image capturing device 3 is camera equipment including image pickupdevices like CCDs (Charged-Coupled Devices) or CMOSs (ComplementaryMetal-Oxide-Semiconductors) and a lens. The image capturing device 3 isa fixed camera, for example. Further, the image capturing device 3 isdesired to have the AGC (Automatic Gain Control) function. AGC isvariable control for keeping the output at a constant level by raisingsensitivity when the input signal is weak (field illuminance is low) andlowering the sensitivity when the input signal is strong (the fieldilluminance is high). The image capturing device 3 provides the imageprocessing device 1 with the captured image.

FIG. 4 is a flowchart showing a captured image region dividing operationand an image processing target region selection operation performed bythe image processing device 1.

First, the image processing device 1 resets the image processing regioninformation 212 in the storage device 210 (step S11). Namely, the imageprocessing device 1 makes the storage device 210 store informationindicating that there is no image processing region information 212.

Subsequently, a captured image acquired by the image capturing device 3is taken into the image processing device 1 by the input interface 30,temporarily held in the memory 220, and thereafter sent to the imageregion calculation unit 110. The image region dividing unit 111 of theimage region calculation unit 110 performs a process of detecting edgeparts in the captured image having great image density difference (i.e.,outlines in the image) and divides the image into n (n≥2) image regionsbased on edge information indicating the edge parts (step S12). Namely,the image region calculation unit 110 performs a dividing process fordividing the captured image into image regions having high monitoringpriority and image regions having low monitoring priority, for example.

Subsequently, the texture measurement unit 112 of the image regioncalculation unit 110 measures a first texture amount in each of theimage regions obtained by the dividing. The first texture amount ismeasured as variance indicating variation (i.e., luminance variation) ina pixel value (i.e., luminance value) in each of the image regionsobtained by the dividing (step S13).

Subsequently, the image processing region selection unit 113 of theimage region calculation unit 110 judges whether or not the firsttexture amount (i.e., the luminance variation) of each of the imageregions obtained by the dividing is less than or equal to thepredetermined texture threshold value 211 (step S14).

The image processing region selection unit 113 of the image regioncalculation unit 110 selects, from the image regions obtained by thedividing, a region whose first texture amount is less than or equal tothe texture threshold value 211 as an image processing target region,and previously stores position information indicating the position ofthe image region selected as the image processing target region in thecaptured image in the storage device 210 as the image processing regioninformation 212 (step S15).

The processing of the steps S13 to S15 is repeated until the processingis performed on all the image regions obtained by the dividing (stepS16). When the first texture amount of every image region obtained bythe dividing exceeds the texture threshold value 211, “nonexistence”information indicating that there exists no image processing targetregion is stored as the image processing region information 212.

In the first embodiment, a region having high monitoring priority is aground surface part including the road 301, whereas the monitoringpriority of the airspace is low. The image processing region selectionunit 113 selects an image region in which the first texture amount issmall (i.e., the luminance variation is small), like the airspace, asthe image processing target region.

Incidentally, the process by the image region calculation unit 110 doesnot have to be executed constantly; it is permissible if the process isexecuted before starting the monitoring operation, periodically(intermittently) during the monitoring operation, or when a changeoccurred in the monitoring region, for example, that is, atpredetermined time points. This makes it possible to avoid placing anexcessive calculation load on the image processing device 1. Further, byexecuting the process of the image region calculation unit 110 when thefield illuminance is high (bright), accuracy of the region dividing canbe increased since the difference in the first texture amount amongimage regions in the captured image is large.

FIG. 5 is a diagram showing an example of a divided region imageobtained by dividing the captured image 300 into two image regions. Inthe example of FIG. 5 , the divided region image 310 has been dividedinto two image regions: an image region 330 in which the first textureamount is large and an image region 320 in which the first textureamount is small (corresponding to the airspace in FIG. 3 ). In thiscase, the image processing target region is the image region 320 inwhich the first texture amount is small (i.e., the luminance variationis small).

FIG. 6 is a flowchart showing an operation of judging whether the imageprocessing should be applied to each image processing target region ornot. First, the image region readout unit 120 reads out the imageprocessing region information stored in the storage device 210 (stepS21). The image region readout unit 120 judges whether or not the imageprocessing region information 212 exists in the storage device 210 (stepS22), and ends the process without executing the image processing whenno image processing region information 212 exists (NO in the step S22).

When the image processing region information 212 exists (YES in the stepS22), the luminance judgment unit 130 calculates average luminance ofthe image processing target region in the captured image based on theimage processing region information 212 (step S23).

Subsequently, the luminance judgment unit 130 judges whether or not thecalculated average luminance is less than or equal to the luminancejudgment threshold value 213 previously stored in the storage device 210(step S24). When the average luminance of the image processing targetregion is less than or equal to the luminance judgment threshold value213, the luminance judgment unit 130 judges that the field illuminanceis low (dark) and thus image capture sensitivity has been set high bythe AGC function of the image capturing device 3, noise is likely tooccur in the captured image, and the image processing is necessary. Incontrast, when the average luminance of the image processing targetregion is higher than the luminance judgment threshold value 213, theluminance judgment unit 130 judges that the field illuminance is high(bright) and there is no need to execute the image processing such asnoise correction.

When the average luminance is judged to be less than or equal to theluminance judgment threshold value 213 in the step S24 (YES in the stepS24), the texture judgment unit 140 calculates a second texture amount(i.e., luminance variation) of the image processing target region of thecurrent captured image, that is, the monitoring target captured imageacquired by the image capturing device 3 (step S25) and compares thecalculated second texture amount with the texture judgment thresholdvalue 214 as the second threshold value previously stored in the storagedevice 210 (step S26). Incidentally, while the second texture amount(i.e., luminance variation) may be obtained as the variance of the pixelvalues in the image processing target region, it is desirable to obtainthe second texture amount as a variation coefficient (i.e., standarddeviation/mean value) of the pixel values in the image processing targetregion which is independent of the absolute values of the pixel valuesin the image processing target region.

When the second texture amount (i.e., luminance variation) is largerthan or equal to the texture judgment threshold value 214 (YES in thestep S26), the image processing unit 150 judges that the noise in theimage processing target region is high and executes the image processingfor correcting (reducing) the noise (step S27). When the second textureamount (i.e., luminance variation) is less than the texture judgmentthreshold value 214 (NO in the step S26), the image processing unit 150judges that the noise in the image processing target region is low andjudges that the image processing for correcting (reducing) the noise isunnecessary. The processing of the steps S23 to S27 is executed for allthe image processing target regions (step S28).

The image after the image processing that has been generated as above isoutputted to the display device such as a display via the outputinterface 40. Incidentally, the image processing mentioned here meansprocessing for reducing (correcting) noise occurring in the image. Theimage processing can be processing by use of a smoothing filter, forexample. However, the method of the image processing is not limited tosuch methods; the image processing can be other processing as long as itis processing for reducing noise occurring in the image.

(1-3) Effect

In the captured image 300 shown in FIG. 3 , when the field illuminanceis low (dark) like in the nighttime, noise is likely to occur in thecaptured image, especially in the airspace far from the image capturingdevice 3 in the captured image 300. Even though an observer wants to seethe lower part of the screen (road 301, vehicle) as the region of highmonitoring priority, the whole screen becomes an image that is hard tosee due to the influence of the noise occurring in the upper part of thescreen (airspace) having low monitoring priority. Especially when thecaptured image 300 is motion video, the noise is perceived as flickeringof the video and the captured image 300 becomes an image (video) that isstill harder to see. With the image processing device 1 according to thefirst embodiment, the captured image acquired by the image capturingdevice 3 as a monitoring camera is divided into a plurality of imageregions and the noise correction processing is performed on regions oflow monitoring priority, by which a monitoring image that is easy to seecan be obtained since the noise in the regions of low monitoringpriority can be reduced and the image blurring due to the influence ofthe noise correction does not occur in regions of high monitoringpriority.

Further, the calculation load on the image processing device 1 can bereduced since the image processing device 1 is configured so that theregion judgment process needing image processing is executed beforestarting the monitoring operation, periodically (intermittently) duringthe monitoring operation, or when a change occurred in the monitoringregion.

Furthermore, the accuracy of the judgment on whether the noisecorrection processing is necessary or not increases since the judgmenton whether the noise correction processing is necessary or not is madebased on the average luminance and the luminance variation (variationcoefficient) in the region of low monitoring priority that is the noisecorrection processing target. Accordingly, unnecessary noise correctionis not carried out and the image quality degradation due to theunnecessary image processing can be prevented.

(2) Second Embodiment (2-1) Configuration

In the first embodiment, the description is given of a case where thesubject that should be monitored is situated outside the imageprocessing target regions. In a second embodiment, a description will begiven of an operation of image processing (i.e., noise correctionprocessing) in a case where a subject as one of monitoring targets issituated in an image processing target region.

FIG. 7 is a block diagram schematically showing a configuration of animage processing device 2 according to the second embodiment. In FIG. 7, each component identical or corresponding to a component shown in FIG.1 is assigned the same reference character as in FIG. 1 . The imageprocessing device 2 is a device capable of executing an image processingmethod according to the second embodiment. The image processing device 2can also be a computer capable of executing an image processing programaccording to the second embodiment.

As shown in FIG. 7 , the image processing device 2 includes a controlunit 10 a, a storage unit 20 a, the input interface 30, the outputinterface 40, and a subject position information acquisition unit 50that acquires real space position information on the subject. Thecontrol unit 10 a includes the image region calculation unit 110, theimage region readout unit 120, the luminance judgment unit 130, thetexture judgment unit 140, a subject position detection unit 160 thattransforms a subject position in the real space into a position in acamera image captured by the image capturing device 3, a mask processingunit 170 that determines a region in the image processing target regionon which the image processing is not performed, and the image processingunit 150. The storage unit 20 a includes one or more storage devices 210and one or more memories 220. The storage device 210 a stores, forexample, a predetermined texture threshold value 211, the imageprocessing region information 212, a predetermined luminance judgmentthreshold value 213, a predetermined texture judgment threshold value214, and camera installation information 215 indicating installationposition information on the image capturing device 3.

(2-2) Operation

The operation of the image processing device 2 according to the secondembodiment until the selection of the image processing target regions isthe same as that in the first embodiment (i.e., the operation shown inFIG. 4 ).

FIG. 8 is a flowchart for explaining the operation of the subjectposition detection unit 160, the mask processing unit 170 and the imageprocessing unit 150 of the image processing device 2 according to thesecond embodiment.

When the texture judgment unit 140 judges that the image processing(i.e., noise correction processing) on the image processing targetregion is necessary, the subject position detection unit 160 reads outthe camera installation information 215 on the image capturing device 3(i.e., camera) previously stored in the storage device 210 and reads outposition information on the subject from the subject positioninformation acquisition unit 50 (steps S31 and S32). Here, the camerainstallation information 215 includes the field angle and the focallength of the camera, the image size (the number of pixels) of thecamera, and coordinate information indicating the camera position in thereal space (world coordinate system). The position information on thesubject includes coordinate information indicating the subject positionin the real space. The camera installation information 215 is previouslyacquired when the image capturing device 3 is installed. The subjectposition information sent from the subject position informationacquisition unit 50 is, for example, GPS information (e.g., latitude,longitude, and altitude) acquired by a GPS (Global Positioning System)module mounted on the subject and transmitted from a transmitterprovided on the subject. In this case, the subject position informationacquisition unit 50 includes a GPS receiver. The subject in the secondembodiment is, for example, an unmanned aircraft (e.g., drone) equippedwith a GPS module.

Subsequently, the subject position detection unit 160 transforms thesubject position in the real space coordinate system into a coordinateposition in the image as viewed from the image capturing device 3 basedon the subject position information and the camera installationinformation 215 (step S33). Specifically, the subject position detectionunit 160 first transforms the subject position in the real spacecoordinate system into that in a camera coordinate system and thentransforms the position into coordinates in the image captured by theimage capturing device 3. The camera coordinate system is, for example,a coordinate system in which the lens center of the image capturingdevice 3 being a camera is placed at the origin. The subject positioninformation (world coordinate system) can be transformed into that inthe camera coordinate system based on the camera installationinformation 215 (the lens position coordinates, the direction of theoptical axis, etc. in the world coordinate system). Subsequently, thesubject position detection unit 160 transforms the subject positioncoordinates, after the transformation into the camera coordinate system,into camera image coordinates.

FIG. 9 is an explanatory diagram showing the process of transforming thesubject coordinates in the camera coordinate system into the cameraimage coordinates. In FIG. 9 , C represents the origin (lens center) ofthe camera coordinate system, and the Z-axis represents a straight line(optical axis) passing through the origin C and orthogonal to the lenssurfaces. The notation (x, y, z) represents the camera coordinates ofthe subject Q existing in the real space. A camera image plane 500 issituated at a position that is separate from the origin C of the cameracoordinate system by the focal length f. Coordinate axes U and V of thecamera image plane 500 are respectively in parallel with coordinate axesX and Y of the camera coordinate system. The notation (u, v) representsthe coordinates of the subject q (indicated by an open circle in FIG. 9) on the camera image plane 500. In this case, the coordinates (u, v) ofq are obtained by the following expressions:u=f·(x/z)  (1)v=f·(y/z)  (2)

Accordingly, the position of the subject Q can be indicated on thecamera image if there is the coordinate position (x, y, z) of thesubject Q in the camera coordinate system, that is, the information onthe latitude, the longitude and the height (altitude) of the subject Qin the real space coordinate system. Incidentally, information on thefocal length f is included in the camera installation information 215.

Subsequently, the subject position detection unit 160 judges whether ornot the subject coordinates q(u, v) on the camera image 400 is includedin the image processing target region (step S34). If q(u, v) is includedin the image processing target region (YES in the step S34), the subjectposition detection unit 160 sets a vicinal region of the coordinates (u,v) including the coordinates (u, v), e.g., an m×n rectangular range, asa mask range on which the image processing is not performed (step S35).Incidentally, m and n are integers larger than or equal to 2. In thiscase, the size of the m×n rectangular range as the mask range as theregion on which the image processing is not performed may be determinedbased on a distance between the subject and the image capturing device3. For example, the mask range is set to be small when the distance fromthe subject to the image capturing device 3 is long, and the mask rangeis set to be large when the distance from the subject to the imagecapturing device 3 is short. The distance from the subject to the imagecapturing device 3 can be obtained from the camera installationinformation 215 and the subject position information acquired from thesubject position information acquisition unit 50. Incidentally, theshape of the mask range is not limited to a rectangular shape but canalso be a different shape such as a circular shape.

When q(u, v) is judged not to be included in the image processing targetregion in the step S34, the processing of the step S35 is not executedand the processing of the steps S33 to S35 is executed for the next(another) subject (step S36). When the processing for all the subjectsis finished, the image processing unit 150 performs the noise correctionprocessing as the image processing on the image processing target regionexcluding the region that has been set as the mask range (step S38).

Here, the processing result of the image processing device 2 accordingto the second embodiment will be described below. FIG. 10(a) is adiagram showing a case in the second embodiment where a subject 510exists in the captured image 300 acquired by the image capturing device3 in the nighttime when the field illuminance is low (dark), wherein astate before performing the image processing in the second embodiment isshown. In the second embodiment, the operation until the operation ofthe texture judgment unit 140 selecting the image processing targetregions is the same as that in the first embodiment, and the existenceand the position of the subject 510 are indicated in the region 320(referred to also as “the region in which the first texture amount issmall”) that is the image processing target region shown in FIG. 5 inwhich the first texture amount is less than or equal to the texturethreshold value 211.

Since the subject 510 is situated at a long distance from the imagecapturing device 3, the subject 510 is captured as a minute object inthe captured image 300. If the image processing (i.e., noise correctionprocessing) is performed on the image processing target region 320 inthis state, there is a possibility that the image part of the subject510 is erroneously judged as noise since the region of the subject 510in the captured image 300 is minute, and the subject 510 can disappearfrom the captured image 300 due to the noise reduction process when thesubject 510 is judged as noise. Especially in the nighttime, thepossibility that the subject 510 is judged as noise is high if thesubject 510 has a light emission means such as an illuminating lamp or ablinking light.

FIG. 10(b) is a diagram showing a case where the rectangular range asthe mask range 520 on which the image processing is not performed isdisplayed in superposition on the divided region image 310 afterundergoing the processing by the image processing device 2 according tothe second embodiment. The mask range 520 is situated in the region 320being the image processing target region in which the first textureamount is small and at a position to include the position of the subject510 in the captured image 300, and if the noise correction processing isperformed in this state, the noise correction processing is notperformed on the mask range 520 in which the subject 510 is situated,and thus the subject 510 does not disappear from the captured image 300even if the subject 510 in the captured image 300 is a minute object.Therefore, the observer can visually recognize the subject 510 in thecaptured image 300. Further, an image easy to see for the observer canbe obtained since the noise correction processing is performed on theregion 320 in which the first texture amount is small excluding the maskrange 520 in which noise is likely to occur.

It is also possible to display the outline of the mask range 520 byusing an emphasis display such as colored frame lines. This allows theobserver to more clearly recognize the position of the subject 510.

Incidentally, it has been assumed in the second embodiment that theinformation acquired by the subject position detection unit 160 isposition information indicating the direction and the altitude of thesubject (e.g., the longitude, the latitude, and the altitude). However,in a case where the information acquired by the subject positiondetection unit 160 also includes individual information (e.g.,identification information or airframe information) on the subject, thesize of the subject can be obtained, and thus the size of the mask range520 may be determined based on the size of the subject and the distanceto the subject. For example, when the distance to the subject is L, thenumber of pixels of the captured image in the width direction is w andthe camera field angle of the image capturing device 3 is θ, the size Δdof the subject in units of pixels can be obtained by the followingexpression (3):Δd=(2L/w)·tan(θ/2)  (3)

When the size of the subject in the horizontal direction is W, based onthe expression (3), a condition required for the number m of pixelsnecessary for the mask range in the width direction can be representedby the following expression (4):m>W/Δd  (4)

Similarly, if the size of the subject in the vertical direction isknown, the number of pixels necessary for the mask range in thelongitudinal direction can be determined by calculation similar to theexpression (4).

Further, while it has been assumed in the second embodiment that thesubject position information is acquired from the subject positiondetection unit 160 in regard to each image captured by the imagecapturing device 3, in cases of video shooting, an image capture cycleof the image capturing device 3 and a subject position informationacquisition cycle of the subject position detection unit 160 can differfrom each other. In general, the image capture cycle of the imagecapturing device 3 is shorter than the subject position informationacquisition cycle. In such cases, subject position information acquiredpreviously is used for the determination of the mask range 520 until thenext subject position information is acquired. For example, when theimage capture cycle of the image capturing device 3 is 1/30 [sec] (i.e.,30 images are captured in one second) and the subject positioninformation acquisition cycle of the subject position detection unit 160is 1 [sec] (i.e., 1 image is captured per second), the mask range 520 isdetermined for the first image by using the subject position informationacquired at that time, the mask range 520 determined from the subjectposition information regarding the former image is used for thesubsequent 29 images, and the mask range 520 is newly determined whenthe next subject position information is acquired. In this case, it isdesirable to set the mask range 520 at a sufficient size with a safetymargin so that the next subject position fits in the previously set maskrange 520.

In a case where speed information on the subject is included in thesubject position information or a case where the moving speed of thesubject can be estimated from past subject position information, thesize of the mask range 520 may be determined as follows, for example:When the cycle of acquiring the subject position information is F [sec],the moving speed of the subject is r [m/sec] and the size of the subjectin units of pixels is Δd [m], the distance (the number of pixels) p inthe image for which the subject moves while the subject positiondetection unit 160 acquires the next subject position information can berepresented by the following expression (5):p=r·F/Δd  (5)

FIG. 11 is a diagram for explaining the size of a mask range 530 afterundergoing speed correction in a case where the moving speed of thesubject is known. Assuming that the size of the mask range 520determined first is m in width and n in length, the mask range after thecorrection is m+2p in width and n+2p in length according to theexpression (5). Incidentally, the reason for adding 2p to the originalsize respectively in the width direction and the longitudinal directionis that it is unknown whether the moving direction of the subject willbe upward, downward, leftward or rightward. When the moving direction ofthe subject is known, the mask range 530 after the correction may be setin a size obtained by adding p to the original size in one of theupward, downward, leftward and rightward directions according to themoving direction of the subject.

(2-3) Effect

As described above, with the image processing device 2 according to thesecond embodiment, the noise processing is not performed on the maskrange 520 where the subject 510 is situated, and thus the subject 510does not disappear from the captured image 300 and can be visuallyrecognized in the captured image 300 by the observer even when thesubject 510 in the captured image 300 is minute. Further, an image easyto see for the observer can be obtained since the noise correctionprocessing is performed on the region 320 in which the first textureamount is small excluding the mask range 520 where noise is likely tooccur.

(3) Modification

While a monitoring target is a road in the above embodiments, usageexamples of the image processing devices 1 and 2 are not limited to suchexamples. For example, the image processing devices 1 and 2 can be usedfor the purpose of monitoring an airdrome surface (airport surface,runway surface). In this case, the road in FIG. 3 corresponds to therunway surface, the subject (vehicle) corresponds to an airplane, andthese constitute the region of high monitoring priority. Further, whilethe region of low monitoring priority is the airspace above the airdromesurface, the airplane can also be situated in the airspace in manycases, and thus employing the image processing device 2 makes itpossible to display the subject (airplane) situated in the airspace tobe easy to see while reducing the noise in the airspace as the region inwhich the first texture amount is small. In cases of an airdromesurface, position information on an airplane can be obtained frominformation provided from an airport surveillance radar or the like. Forexample, position information (distance, direction, altitude) and modelinformation on an airplane in the airspace can be obtained byconfiguring the subject position information acquisition unit 50 toacquire SSR (Secondary Surveillance Radar) information and airplaneflight information.

DESCRIPTION OF REFERENCE CHARACTERS

1, 2: image processing device, 3: image capturing device, 10, 10 a:control unit (processor), 20, 20 a: storage unit, 30: input interface,40: output interface, 50: subject position information acquisition unit,110: image region calculation unit, 111: image region dividing unit,112: texture measurement unit, 113: image processing region selectionunit, 120: image region readout unit, 130: luminance judgment unit, 140:texture judgment unit, 150: image processing unit, 160: subject positiondetection unit, 170: mask processing unit, 210: storage device, 220:memory, 211: texture threshold value (first threshold value), 212: imageprocessing region information, 213: luminance judgment threshold value,214: texture judgment threshold value (second threshold value), 215:camera installation information, 300: captured image, 301: road, 310:divided region image, 320: region in which first texture amount issmall, 330: region in which first texture amount is large, 500: cameraimage plane, 510: subject, 520: mask range, 530: mask range aftercorrection.

What is claimed is:
 1. An image processing device that performs imageprocessing for reducing image noise, comprising: a processor to executea program; and a memory to store the program which, when executed by theprocessor, performs processing including dividing a first captured imageinto a plurality of image regions and determining an image processingtarget region based on a first texture amount indicating luminancevariation in the plurality of image regions; measuring the first textureamount in a second captured image, said second captured image havingbeen captured after said first captured image and including the imageprocessing target region; judging whether the image processing on thesecond captured image is necessary or not based on luminance of theimage processing target region in the second captured image; calculatinga second texture amount indicating luminance variation in the imageprocessing target region for which the image processing is judged to benecessary in said judging based on the luminance, and judging whetherthe image processing should be performed on the image processing targetregion or not based on the second texture amount; and performing theimage processing on the image processing target region on which saidjudging based on the second texture amount is that the image processingshould be performed.
 2. The image processing device according to claim1, wherein said calculating includes selecting an image region includedin the plurality of image regions and in which the first texture amountis less than or equal to a predetermined first threshold value as theimage processing target region.
 3. The image processing device accordingto claim 1, wherein said judging based on the luminance includes judgingthat the image processing on the current captured image is necessarywhen the luminance of the image processing target region is less than orequal to a predetermined luminance judgment threshold value.
 4. Theimage processing device according to claim 1, wherein said judging basedon the second texture amount includes judging that the image processingon the second captured image is necessary when the second texture amountof the image processing target region is larger than or equal to apredetermined second threshold value.
 5. The image processing deviceaccording to claim 1, wherein the first texture amount is varianceindicating variation of pixel values in each of the plurality of imageregions.
 6. The image processing device according to claim 1, whereinthe luminance of the image processing target region is a mean value ofluminance values in the image processing target region.
 7. The imageprocessing device according to claim 1, wherein the second textureamount is a variation coefficient of pixel values in the imageprocessing target region.
 8. The image processing device according toclaim 1, wherein the memory stores image processing region informationindicating the image processing target region previously determined bysaid calculating.
 9. The image processing device according to claim 1,wherein the memory stores position information on a camera that acquiresthe first and second captured images, the program which, when executedby the processor, performs processing including acquiring positioninformation on a subject; transforming position coordinates of thesubject in a real space into position coordinates in a camera imagespace of the camera; and setting a region including the positioncoordinates of the subject as a mask range in the camera image spacewhen the subject is situated in the image processing target region inthe camera image space, and the image processing is performed on theimage processing target region excluding the mask range.
 10. The imageprocessing device according to claim 9, wherein said setting includesacquiring size information indicating a size of the subject and distanceinformation indicating a distance from the camera to the subject fromthe second captured image, and determining a size of the mask rangebased on the size information and the distance information.
 11. Theimage processing device according to claim 9, wherein said settingincludes acquiring a moving speed of the subject from the secondcaptured image, and determining a size of the mask range based on themoving speed, an acquisition cycle of acquiring the positioninformation, and an image capture cycle of the camera.
 12. The imageprocessing device according to claim 9, wherein the mask range isdisplayed by using an emphasis display.
 13. An image processing methodof performing image processing for reducing image noise, comprising:dividing a first captured image into a plurality of image regions anddetermining an image processing target region based on a first textureamount indicating luminance variation in the plurality of image regions;measuring the first texture amount in a second captured image, saidsecond captured image having been captured after said first capturedimage and including the image processing target region; judging whetherthe image processing on the second captured image is necessary or notbased on luminance of the image processing target region in the secondcaptured image; calculating a second texture amount indicating luminancevariation in the image processing target region for which the imageprocessing is judged to be necessary in said judging based on theluminance, and judging whether the image processing should be performedon the image processing target region or not based on the second textureamount; and performing the image processing on the image processingtarget region on which said judging based on the second texture amountis that the image processing should be performed.
 14. A non-transitorycomputer-readable storage medium storing an image processing programthat causes a computer to perform image processing for reducing imagenoise, wherein the image processing program causes the computer toexecute processing comprising: dividing a first captured image into aplurality of image regions and determining an image processing targetregion based on a first texture amount indicating luminance variation inthe plurality of image regions; measuring the first texture amount in asecond captured image, said second captured image having been capturedafter said first captured image and including the image processingtarget region; judging whether the image processing on the secondcaptured image is necessary or not based on luminance of the imageprocessing target region in the second captured image; calculating asecond texture amount indicating luminance variation in the imageprocessing target region for which the image processing is judged to benecessary in said judging based on the luminance, and judging whetherthe image processing should be performed on the image processing targetregion or not based on the second texture amount; and performing theimage processing on the image processing target region on which saidjudging based on the second texture amount is that the image processingshould be performed.